AdaCB: An Adaptive Gradient Method with Convergence Range Bound of Learning Rate

Adaptive gradient descent methods such as Adam, RMSprop, and AdaGrad achieve great success in training deep learning models. These adaptively change the rates, resulting a faster convergence speed. Recent studies have shown their problems include extreme non-convergence issues, well poor generalization. Some enhanced variants been proposed, AMSGrad, AdaBound. However, performances of these alternatives are controversial some drawbacks still occur. In this work, we proposed an optimizer called AdaCB, which limits rates Adam range bound. The bound is determined by LR test, then two functions designed to constrain tend constant value. To evaluate our method, carry out experiments on image classification task, three models including Smallnet, Network IN Network, Resnet trained CIFAR10 CIFAR100 datasets. Experimental results show that method outperforms other optimizers datasets with accuracies (82.76%, 53.29%), (86.24%, 60.19%), (83.24%, 55.04%) Resnet, respectively. also indicate maintains speed, like adaptive methods, early stage achieves considerable accuracy, SGD (M), at end.